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Foreword 


This  technical  note  describes  work  conducted  as  part  of  the  Navy  Personnel  Research  and 
Development  Center's  Complex  Cognitive  Skills  (COPE)  project.  The  COPE  project  is  part  of  the 
Navy  Exploratory  Development  Program,  Technology  Program  Element  0602233N,  Work  Unit 
RM33T23.08,  sponsored  by  the  Office  of  Naval  Research. 

The  first  part  of  the  COPE  project  investigated  methods  for  analyzing  cognitive  skills  in 
technical  tasks;  this  work  was  reported  in  a  paper  submitted  for  publication  (Randel,  Pugh,  &  Reed, 
1995).  For  this  technical  note,  we  have  selected  several  of  the  methods  that  revealed  expert/novice 
differences  and  demonstrated  how  these  methods  could  be  used  to  conduct  a  cognitive  task  analysis 
(CTA)  in  domains  involving  decision  making  tasks. 

The  authors  wish  to  acknowledge  the  contribution  of  John  S.  Schuler  for  his  initial  work  and 
ideas  on  the  information  flow  model. 


J.  C.  McLACHLAN 
Director,  Classroom  and  Afloat  Training 
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Summary 


Background 

Analysis  of  the  cognitive  factors  involved  in  a  complex  task  should  provide  improved  training 
and  assessment.  Cognitive  task  analysis  (CTA)  enhances  traditional  task  analysis  procedures  by 
analyzing  the  thought  processes  of  performers  while  they  complete  a  task.  One  way  of  identifying 
the  cognitive  skills  needed  to  perform  a  task  well  is  to  compare  expert  with  novice  understanding 
of  the  task.  When  experts  are  found  to  differ  from  novices  on  the  cognitive  aspects  of  a  task,  this 
input  can  be  used  to  improve  training  design.  Earlier  work  for  this  project  examined  the  Electronic 
Warfare  (EW)  operator  task  for  differences  between  novices  and  experts  in  several  areas  of 
cognitive  functioning.  Many  of  these  measures  revealed  differences  between  novices  and  experts. 

Purpose 

The  purpose  of  this  report  is  to  provide  a  guide  to  those  who  wish  to  conduct  a  cognitive  task 
analysis  of  a  technical  task  that  emphasizes  decision  making.  It  is  meant  for  those  who  are  familiar 
with  the  concept  of  task  analysis  and  wish  to  include  cognitive  aspects  of  their  task  in  this  analysis. 
It  does  not  replace  traditional  task  analysis,  but  adds  the  cognitive  components  to  it. 

Method  and  Results 

We  have  selected  several  of  the  measures  for  which  we  found  differences  between  experts  and 
novices  and  have  detailed  the  procedures  for  using  these  methods  to  conduct  a  CTA  for  domains 
which  involve  critical  decision  making  tasks  in  naturalistic  settings.  The  cognitive  task  analysis 
methods  that  we  describe  are  (1)  development  of  a  task  process  model,  (2)  development  of  an 
information  flow  model,  (3)  misconceptions  analysis,  and  (4)  structural  knowledge  analysis. 

For  each  of  these  methods  we  describe  how  information  is  gathered  (knowledge  elicitation), 
how  it  is  represented  in  a  format  that  can  be  later  used  in  designing  training  (knowledge 
representation),  and  how  the  knowledge  can  be  used  in  developing  training. 

Conclusions 

The  methods  described  here  are  appropriate  for  conducting  a  cognitive  task  analysis  for  a 
complex  decision  making  task.  Any  of  the  methods  could  be  used  individually  or  in  combination. 

Recommendations 

We  recommend  that  the  cognitive  task  analysis  methods  described  here  be  used  in  developing 
or  revising  instruction  for  complex  decision  making  tasks.  These  methods  could  be  used  in 
conjunction  with  traditional  task  analysis. 
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Introduction 


Background 

The  modem  work  environment  requires  skilled  operators  to  perform  tasks  on  complex  systems 
in  an  efficient,  cost-effective,  and  safe  manner.  Operator  training  for  such  tasks  as  those  performed 
by  air  traffic  controllers,  radar  operators,  and  electronic  warfare  technicians  often  emphasize  the 
learning  of  facts  and  procedures.  Recent  developments  in  cognitive  science  indicate  that  an 
analysis  of  the  cognitive  factors  involved  in  a  skill  should  provide  improved  training  and 
assessment  (DuBois  &  Shalin,  1995;  Gordon,  Schmierer,  &  Gill,  1993;  Hadl,  Gott,  &  Pokomy, 
1990).  To  understand  how  an  operator  conceptualizes  a  task,  we  need  to  use  methods  which  reveal 
not  only  the  behaviors  that  are  being  performed  but  also  the  decisions  that  are  made  and  the 
decision  makers'  reasons  for  these  actions. 

The  traditional  method  for  analyzing  a  task  is  to  decompose  it  into  subtasks,  skills,  and 
knowledge.  Cognitive  task  analysis  (CTA)  improves  this  process  by  including  an  analysis  of  the 
thought  processes  of  performers  while  they  complete  a  task.  One  of  the  earliest  attempts  to  perform 
CTA  (Greeno,  1976)  demonstrated  the  use  of  procedural  representations  and  semantic  networks  for 
a  cognitive  task  analysis  of  topics  taught  in  the  classroom.  More  recently,  others  have  applied 
cognitive  analyses  to  tasks  performed  on  the  job;  for  example,  the  strategic  knowledge  used  in 
solving  a  troubleshooting  task  was  considered  along  with  declarative  and  procedural  knowledge 
(Hall  et  al.,  1990).  For  analyzing  a  complex  operator  task  involving  decision  making,  no  adequate 
CTA  method  was  available. 

Although  the  methods  being  introduced  in  this  paper  are  intended  to  be  relatively  simple  and 
economical,  it  should  be  recognized  that  cognitive  task  analysis  can  raise  the  cost  of  task  analysis 
so  the  cost-benefit  of  such  analysis  should  be  considered.  Because  thorough  cognitive  analysis  may 
require  the  skills  of  highly  trained  professionals  at  its  present  stage  of  development  (Lesgold  et  al., 
1986;  Modrick,  1992;  Redding,  1989),  consideration  should  be  given  to  the  appropriateness  of 
using  cognitive  task  analysis  for  any  task.  Gott  (1994)  suggested  that  CTA  should  be  used  when  a 
task  is  complex,  when  it  is  difficult  to  learn  because  action  goes  on  in  the  head  of  a  performer,  and 
when  tasks  are  not  presequenced;  that  is,  the  task  is  dynamic,  unstable,  and  ill-structured.  These 
requirements  describe  the  decision  making  task  of  the  electronic  warfare  technician  (EW). 

We  conducted  an  experimental  investigation  of  methods  for  analyzing  cognitive  skills  in  a 
technical  task  involving  decision  making  (Randel,  Pugh,  &  Reed,  1995).  One  way  of  identifying 
the  cognitive  skills  needed  to  perform  a  task  well  is  to  compare  expert  with  novice  understanding 
of  the  task.  Experts  and  novices  were  identified  on  the  basis  of  their  performance  on  a  technical 
task  as  measured  by  a  performance  assessment  instrument.  The  operator  task  performed  by  the 
electronic  warfare  technician  (EW)  was  examined  for  expert-novice  differences  in  several  areas  of 
cognitive  functioning. 

EW  operators  working  on  ships  use  computer  automated  equipment  to  help  detect  and  identify 
potentially  hostile  radar  systems.  The  EW  operators  extract  clues  as  to  the  possible  threat  of  a  radar 
by  observing  on  a  display  screen  a  diagram  of  the  positions  of  all  the  emitters  in  the  area,  by 
looking  at  the  computer  readout  of  certain  parameters  of  the  signal,  and  by  listening  to  an  auditory 
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representation  of  the  signal.  Along  with  the  computer  data,  the  EW  uses  information  gathered  from 
other  watchstanders  aboard  ship  and  published  data  to  make  his  assessment  of  the  situation.  The 
EW  informs  higher  authority  of  the  nature  of  threat  signals  and  recommends  appropriate  actions 
including  countermeasures.  Further  details  on  the  EW  job  and  experimental  study  of  EW  expert 
and  novice  differences  can  be  found  in  Randel  et  al.  (1995).  The  present  technical  note  takes  some 
of  the  cognitive  task  analysis  techniques  investigated  and  shows  how  they  can  be  used  to  analyze 
a  technical  task.  Here,  the  EW  is  used  as  an  example  only;  it  is  not  necessary  to  understand  the  EW 
job  completely  to  follow  these  procedures. 

Purpose 

The  purpose  of  this  report  is  to  provide  a  guide  to  those  who  wish  to  conduct  a  cognitive  task 
analysis  of  a  technical  task  that  emphasizes  decision  making.  It  is  meant  for  those  who  are  familiar 
with  the  concept  of  task  analysis  and  wish  to  include  cognitive  aspects  of  their  task  in  this  analysis. 
It  does  not  replace  traditional  task  analysis  but  adds  the  cognitive  components  to  it. 

For  this  report,  we  have  selected  several  of  the  measures  for  which  we  found  differences 
between  experts  and  novices  and  which  would  generalize  to  other  domains.  We  have  detailed  the 
procedures  for  using  these  methods  to  conduct  a  CTA  for  domains  which  involve  critical  decision 
making  tasks  in  naturalistic  settings.  We  have  taken  the  methods  investigated  experimentally  and 
indicated  how  they  can  be  used  in  domains  other  than  the  EW. 

Method  and  Results 

The  methods  that  we  will  describe  here  are: 

1.  Development  of  a  task  process  model. 

2.  Development  of  an  information  flow  model. 

3.  Misconceptions  analysis. 

4.  Structural  knowledge  analysis. 

For  each  of  these  methods  we  will  indicate  (1)  how  knowledge  is  elicited,  (2)  how  it  is  represented, 
and  (3)  how  it  can  be  used  in  training. 

Development  of  a  Task  Process  Model 

One  aspect  of  cognitive  task  analysis  is  the  development  of  a  model  of  a  task  that  requires 
decision  making.  By  model  we  mean  a  graphic  or  schematic  representation  of  the  task.  The  model 
is  developed  from  observation  and  questioning  of  the  task  performers  either  while  they  are 
performing  the  task,  or,  if  the  task  would  be  disrupted  by  the  questioning,  after  the  task  has  been 
completed.  The  objective  is  to  develop  a  model  of  the  task  that  can  be  used  to  organize  training. 
Each  part  of  the  model  can  be  taught  in  training  and  anything  taught  can  be  related  back  to  the  task 
model.  Training  that  is  organized  around  a  task  will  be  seen  as  relevant  to  the  actual  job  to  be 
performed  and  as  a  topic  is  taught,  its  use  in  the  actual  job  can  be  seen. 
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In  the  following  three  sections  on  the  development  of  a  task  process  model,  we  will  first  discuss 
how  information  about  task  performance  is  obtained  to  build  the  model  (knowledge  elicitation), 
and  then  explain  the  form  in  which  the  knowledge  is  documented  (knowledge  representation). 
Finally,  we  consider  how  this  model  can  be  applied  to  training. 

Knowledge  Elicitation  for  a  Task  Process  Model 

The  purpose  of  knowledge  elicitation  is  to  gather  information  on  how  the  task  is  performed  so 
that  this  information  can  be  represented  as  a  model.  The  following  steps  explain  how  to  gather  this 
information. 

1.  Obtain  an  understanding  of  the  job.  The  more  complex  the  job,  the  more  you  will  need 
to  know  about  it  to  understand  what  a  task  performer  or  subject  matter  expert  (SME)  is  telling  you 
during  questioning  in  an  interview.  This  understanding  can  be  achieved  in  a  number  of  ways. 

a.  Collect  and  analyze  documentation  relevant  to  the  task  to  get  an  overview  of  the  job 
being  performed.  Documentation  might  include:  training  documents,  fleet  training  requirements, 
and  occupational  standards. 

b.  Observe  any  available  training  exercises  or  actual  job  performance. 

c.  List  the  major  high  level  tasks  (e.g.  for  the  EW,  four  major  high  level  tasks  were 
identified:  emitter  identification,  communication,  countermeasures,  system  operation). 

2.  Select  subject  matter  experts  (SMEs).  If  possible,  select  SMEs  on  the  basis  of  an  opera¬ 
tional  performance  measure.  If  a  performance  measure  is  available,  both  experts  and  novices 
could  be  studied  for  comparison. 

3.  Conduct  interviews.  The  interviews  have  two  purposes:  (a)  to  develop  a  model  of  the 
task,  and  (b)  to  identify  critical  components  of  the  decision  making  process.  The  output  from 
both  of  these  elements  can  be  used  in  the  design  and  development  of  training.  The  interviews 
should  be  conducted  in  two  stages;  the  first  stage  allows  you  to  develop  a  preliminary  model  of 
the  task  which  is  then  analyzed  for  completeness.  From  this  analysis,  questions  are  developed  for 
the  second  stage  interview.  The  second  stage  interview  ensures  the  accuracy  of  the  preliminary 
model  and  verifies  the  decision  information  collected.  The  second  stage  interview  also  enables 
elaboration  of  the  task  model  to  an  appropriate  size  (usually  no  more  than  two  pages)  that  can  be 
used  in  training. 

a.  Stage  1  Interview.  The  purpose  of  this  interview  is  to  develop  a  preliminary  model  of 
the  task  and  to  identify  areas  in  the  model  where  additional  information  is  needed.  This  interview 
can  be  conducted  after  a  simulation  exercise,  after  a  training  exercise,  or,  if  these  are  not  available, 
from  the  description  of  a  critical  incident  that  has  occurred  in  the  past.  If  a  simulation  exercise  is 
used,  the  simulation  should  contain  some  incidents  where  a  moderately  difficult  decision  must  be 
made.  It  should  also  ideally  contain  some  of  the  usual  elements  of  incidents  that  would  normally 
take  place  during  a  period  of  job  performance.  Approximately  three  to  five  interviewees  should  be 
obtained  for  this  stage. 
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Critical  Incident.  Get  interviewees  to  describe  a  critical  incident  from  the  simulation 
or  from  their  experience.  A  critical  incident  should  involve  a  situation  where  error  was  likely  or 
did  in  fact  occur.  Use  the  Critical  Incident  Interview  Guide  (Appendix  A),  which  is  based  on  the 
work  on  naturalistic  decision  making  of  Klein  and  associates  (Calderwood,  Crandall,  &  Klein 
(1987).  For  any  decision  point  that  is  probed,  obtain  information  on  the  cues  involved,  errors  made, 
missing  data,  differences  between  novices  and  experts,  knowledge  used,  decision  options  (if  any), 
and  rules  of  thumb.  If  actual  experience  of  interviewee(s)  is  not  appropriate,  then  probe  for  a 
simulation  exercise  that  was  found  particularly  challenging  during  training.  Probe  for  the  important 
tasks  that  are  performed  on  the  watch  and  which  contribute  to  decision  making  or  problem  solving. 

Preliminary  Model.  From  this  interview  and  the  documentation  analyzed,  draw  a 
preliminary  task  process  model  in  the  form  of  a  flowchart  of  how  each  interviewee  conceives  the 
job.  It  is  particularly  important  to  include  the  decision  points.  Figure  1  is  an  example  of  a  simple 
task  process  model,  with  only  one  decision  point  (diamond  shape)  shown.  Your  preliminary  model 
may  contain  more  steps  than  this. 


Figure  1.  Preliminary  task  process  model. 


Composite  Model.  After  you  have  completed  a  model  for  each  individual,  construct  a 
preliminary  composite  model  based  on  the  elements  which  are  mentioned  most  often.  It  may  be 
necessary  to  include  some  elements  which  are  not  mentioned  frequently,  but  which  were  stated  to 
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be  important.  This  preliminary  model  will  help  you  understand  how  well  you  comprehend  the 
tasks.  Note  where  you  are  uncertain  of  what  is  occurring  in  the  performance  of  the  task,  and  write 
questions  to  ask  about  these  areas  in  the  Stage  2  interview. 

b.  Stage  2  Interview.  These  interviews  would  optimally  involve  an  additional  8-20 
interviewees,  with  some  novices  as  well  as  experts,  if  possible.  These  interviews  make  use  of 
material  gathered  during  Stage  1  interviews.  Additional  questions  will  have  been  created  to  answer 
any  uncertainties  about  the  task  process  model  of  the  task.  An  example  of  such  questions  used  in 
analyzing  the  EW  task  are  given  in  Appendix  B.  Present  the  scenario  followed  by  the  interview, 
using  the  Critical  Incident  Interview  Guide. 

Knowledge  Representation  for  a  Task  Process  Model 

The  knowledge  obtained  by  this  method  will  be  represented  as  a  model  of  the  task.  For  each 
individual  interviewed  in  Stage  2,  construct  a  model  of  the  task  the  same  way  you  did  for  the 
preliminary  models.  Use  the  information  obtained  from  the  new  questions  constmcted  from  the 
first  interviews  and  asked  in  Stage  2.  Then  construct  a  new  composite  model  of  the  task  for  the 
experts.  If  novices  were  interviewed,  a  separate  model  could  be  constructed  to  contrast  with  the 
expert  model. 

An  example  of  such  a  process  task  model  is  contained  in  Appendix  C  for  the  EW.  It  was  kept 
to  two  pages  because  any  greater  detail  at  this  time  would  make  it  too  complex  to  be  used  to  guide 
instruction.  While  the  model  is  of  necessity  a  simplified  outline  of  the  task  to  be  performed,  it  could 
be  further  extended  to  include  the  elements  that  make  up  any  of  the  boxes.  Note  that  the  first  row 
of  boxes  in  Appendix  C  displays  five  subtasks  that  need  to  be  performed.  The  order  of  these  five 
subtasks  may  vary  depending  on  the  situation,  and  this  will  need  elaboration  during  instruction. 

After  the  model  is  constructed  it  should  be  validated  by  presenting  it  to  other  knowledgeable 
SMEs  such  as  instructors  and  people  with  recent  experience  on  the  task.  On  the  basis  of  input  from 
this  group,  the  model  may  be  modified.  However,  once  modifications  are  made,  the  model  should 
be  again  presented  to  additional  SMEs  to  ensure  there  is  agreement  on  the  modifications. 

Training 

The  knowledge  represented  by  the  task  model  can  be  used  as  an  organizing  framework  to  train 
people  to  perform  the  job.  During  instruction,  any  of  the  boxes  could  be  expanded  to  greater 
detail.  This  model  should  not  be  considered  static  and  can  be  modified  as  situations  or  preferences 
change.  However,  if  it  is  displayed  in  a  prominent  place  in  the  classroom,  perhaps  on  a  white  board 
so  it  can  be  modified,  each  module  of  instruction  can  be  related  to  the  overall  task. 

If  a  novice  model  was  constructed,  differences  between  this  model  and  the  expert  model  could 
be  contrasted.  The  novice  model  would  indicate  how  the  novice  generally  thinks  about  the  task. 
The  curriculum  developer  could  use  this  information  to  determine  how  it  differs  from  the  expert 
and  then  attempt  to  bring  the  novice  gradually  to  the  more  expert  way  of  thinking. 

On  the  basis  of  the  expert  model,  realistic  situations  requiring  difficult  decision  making  could 
be  developed  to  represent  each  of  the  decision  areas  in  the  model.  Experts  could  be  interviewed  for 
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cases  involving  these  situations,  and  these  could  then  be  used  in  training.  Using  the  model  as  the 
basis  for  these  situations  could  ensure  that  all  important  areas  are  covered. 

Development  of  an  Information  Flow  Model 

In  order  to  make  informed  decisions,  it  is  necessary  to  gather  information  from  a  variety  of 
sources,  and  it  is  often  necessary  to  pass  information  on  to  others  in  the  decision  making  process. 
Information  flow  refers  to  the  process  of  gathering  information,  relating  it  to  other  information,  and 
transmitting  acquired  information  to  appropriate  recipients.  To  understand  how  the  flow  of 
information  is  conceptualized,  the  task  performer  can  be  asked  to  draw  or  diagram  the  path  for 
information  flow.  Data  from  both  experts  and  novices  can  be  compared  and  used  to  guide  training. 

In  the  following  three  sections  on  the  development  of  an  information  flow  model,  we  will  first 
discuss  how  the  information  flow  model  is  obtained  (knowledge  elicitation),  how  the  knowledge 
is  documented  (knowledge  representation),  and  how  this  model  can  be  applied  to  training. 

Knowledge  Elidtation  for  an  Information  Flow  Model 

To  obtain  data  on  a  task  performer’s  mental  representation  of  information  flow,  present  a 
specific  instance  in  which  information  must  be  relayed  and  ask  the  respondent  to  diagram  how 
information  is  communicated  for  this  particular  task.  Tell  them  to  use  boxes  for  each  person  or 
position  and  arrows  to  indicate  the  direction  of  the  information.  For  example,  we  asked  the  EW  to 
diagram  how  information  is  communicated  aboard  ship  and  to  the  battle  group  once  a  new  emitter 
is  detected. 

Knowledge  Representation  for  an  Information  Flow  Model 

Taking  the  information  flow  models  drawn  by  the  individuals,  construct  a  composite  model 
based  on  the  elements  that  are  mentioned  most  often.  Sometimes  this  will  involve  a  judgment  call 
if  something  is  mentioned  less  frequently  but  its  importance  is  emphasized.  An  example  of  a 
composite  information  flow  model  for  the  EW  expert  group  is  given  in  Figure  2.  This  example  is 
given  to  show  the  type  of  diagram  with  boxes  and  arrows  that  might  be  used;  for  this  discussion  it 
is  not  necessary  to  understand  each  of  these  jobs. 

The  point  of  this  diagram  is  to  show  to  whom  the  worker  communicates  directly  and  indirectly. 
K  a  novice  diagram  is  constructed,  comparisons  can  be  drawn  between  the  novice  and  expert 
diagrams.  In  Figure  2,  we  see  that  the  EW  operator  communicates  directly  to  everyone  in  this 
diagram,  even  though  his  supervisor  also  communicates  to  many  of  these  same  people.  The  novice 
diagram  showed  that  the  novice  relied  on  the  supervisor  for  most  of  his  communications  and  did 
not  directly  contact  the  other  workers. 
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Training 

Information  flow  maps  for  any  job  can  be  used  to  teach  students  who  they  should  be 
communicating  with  to  obtain  and  disseminate  information.  Using  information  flow  maps  to 
emphasize  the  need  to  seek  information  could  improve  communications  in  any  job  involving 
several  players.  In  addition,  these  maps  could  be  used  to  develop  a  computer  simulation  to  practice 
selecting  appropriate  sources  of  information  for  different  tactical  situations. 

For  assessment  purposes,  information  flow  maps  could  be  obtained  from  students  at  an 
appropriate  point  in  their  instruction.  The  maps  the  students  draw  can  be  compared  to  those 
obtained  during  the  analysis  phase  of  the  program  or  those  constructed  by  the  instructors.  Different 
maps  may  be  drawn  for  different  tasks  or  situations. 

Misconceptions 

Related  to  decision  making,  and  perhaps  imposing  limits  on  it,  are  misconceptions  that  might 
be  held  in  conceptualizing  a  task.  Feltovich,  Spiro,  and  Coulson  (1989)  studied  the  structure  of 
complex  ideas  and  the  development  of  misconceptions  in  biomedicine.  They  describe  a  pattern  of 
biomedical  misunderstandings  in  which  the  interdependency  of  misconceptions  and  the 
oversimplification  of  concepts  are  two  prominent  features  of  misunderstandings. 

It  seems  reasonable  to  look  for  the  interdependency  of  misconceptions  and  also  to  look  for 
cases  of  oversimplification  on  the  part  of  operators—particularly  in  the  novice  operators.  Larkin, 
McDermot,  Simon,  and  Simon  (1980)  indicate  that  experts  can  see  underlying  causes  because  they 
can  refer  to  the  complex  models  they  have  stored.  Obtaining  comments  and  qualifying  remarks 
from  operators  can  help  us  understand  what  complex  models  aid  them  in  their  decisions. 
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One  way  to  study  misconceptions  is  to  examine  the  heuristics  or  rules  of  thumb  that  operators 
use  in  making  decisions.  Rules  of  thumb  are  often  used  by  experts  to  represent  knowledge  about  a 
task  that  novices  can  follow.  A  rule  of  thumb  can  be  considered  compiled  knowledge  of  the  expert 
presented  in  an  integrated  form.  A  problem  could  arise  when  the  novice  does  not  understand  the 
boundary  conditions  of  the  rule  or  when  it  may  not  apply.  Decision  aids,  of  which  rules  of  thumb 
could  be  considered  an  example,  have  been  shown  to  be  problematical  when  novice  decision 
makers  use  aids  based  on  expert-level  representation  (Ryder,  Zachary,  Zaklad,  &  Purcell,  1994). 

In  the  following  three  sections,  we  describe  a  method  for  studying  misconceptions  in  rules  of 
thumb.  We  discuss  how  the  information  on  misconceptions  is  obtained  (knowledge  elicitation), 
how  the  knowledge  is  documented  (knowledge  representation),  and  how  this  information  can  be 
applied  to  training. 

Knowledge  Elicitation  for  Misconceptions 

To  study  misconceptions  in  rules  of  thumb,  information  about  possible  misconceptions  is 
obtained  through  the  following  three  steps. 

1.  Prepare  the  list  of  rules.  While  discussing  decisions  for  the  task  process  model,  or  inde¬ 
pendently  in  relation  to  a  simulation  or  real  event,  obtain  information  on  rules  of  thumb  used  in 
making  tiie  identified  decision.  With  regard  to  the  decision  being  discussed,  ask  what  advice  or 
rule  of  thumb  the  operator  would  give  to  someone  standing  beside  him  or  her  during  a  decision 
(see  Appendix  B,  item  5f).  You  should  try  to  have  a  list  of  about  12-15  rules  which  workers  have 
said  would  be  good  rules  to  follow  in  realistic  job  situations.  It  would  be  ideal  to  obtain  these 
rules  from  a  range  of  operators  from  novice  to  expert. 

When  preparing  the  rules  of  thumb  to  be  considered  at  the  next  stage  in  the  interview,  look 
for  apparent  contradictions.  Often  the  contradictions  will  be  subtle  and  not  immediately  obvious. 
For  example,  for  the  EW,  the  following  pair  of  rules  were  included:  (a)  Don't  report  an  emitter  as 
a  missile  unless  you  are  sure  it's  a  missile;  (b)  better  to  call  something  a  missile  when  it  is  not  a 
missile  than  the  other  way  around.  There  is  an  apparent  conflict  between  (a)  and  (b);  namely,  don't 
report  something  as  a  missile  unless  you're  sure,  but  if  you're  not  sure  then  call  something  a  missile. 
Be  sure  that  you  are  aware  of  such  contradictions,  but  do  not  pair  them  up  on  your  list.  Prepare  the 
list  of  plausible  rules,  consolidating  the  inputs  as  appropriate. 

2.  Interview  operators  on  each  rule.  Ask  whether  each  rule  is  good  to  follow  and  ask  oper¬ 
ators  to  give  reasons  why  the  rules  are  good  or  bad.  Note  particularly  the  exceptions  that  they 
make,  and  also  note  the  situations  that  they  describe  for  the  application  of  the  rule.  Experts  often 
give  important  exceptions  to  rules,  so  it  is  important  to  note  elaborations  which  tell  when  and  how 
a  rule  is  to  be  applied.  These  comments  may  provide  the  most  important  information. 

3.  Question  apparent  contradictions.  When  you  prepared  the  list  of  rules  you  noted  which 
rules  were  not  in  agreement.  Consider  the  two  rules  that  were  given  as  an  example  above.  If 
someone  agrees  that  both  of  two  contradictory  rules  are  good  to  follow,  then  it  is  important  to  find 
out  how  such  a  contradiction  should  be  resolved.  For  example,  the  subject  may  say  that  when  you 
are  uncertain  that  an  emitter  is  a  missile,  that  you  should  check  with  the  watch  supervisor  first,  or 
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that  it  should  be  reported  as  a  missile  but  that  the  uncertainty  should  be  reported  also.  Thus, 
exceptions  and  elaborations  are  very  important. 

You  can  compare  experts  and  novices  on  the  exceptions  and  elaborations  they  provide.  Experts 
will  often  anticipate  a  conflict  between  rules  and  explain  how  they  apply  to  different  situations 
even  before  you  point  out  particular  contradictions.  Experts  tend  to  see  the  big  picture,  and  so  they 
will  accept  fewer  contradictory  pairs  of  rules  than  novices.  Questioning  experts  on  apparent 
contradictions  can  bring  out  the  tacit  knowledge  that  underlies  expertise.  Making  use  of  this 
expertise  will  help  to  train  students  better  and  more  quickly. 

Knowledge  Representation  for  Misconceptions 

1.  List  the  rules  that  are  considered  useful  in  making  decisions.  Include  the  elaborations  and 
exceptions  to  the  rules  obtained  in  the  interviews.  Specific  examples  of  situations  should  also  be 
recorded. 

2.  If  incompatible  rules  were  found,  list  these  also.  Include  explanations  of  the  circumstance 
under  which  these  rules  would  be  used. 

Training 

The  result  of  carefully  preparing  a  list  of  rules  of  thumb  and  then  probing  these  rules  during 
interviews  is  that  you  have  important  information  to  guide  the  development  of  training.  The  rules 
themselves  are  guides  to  what  workers  actually  use  on  the  job.  Making  these  rules  explicit  along 
with  the  elaborations  and  exceptions  ensures  that  novice  operators  will  know  the  boundary 
conditions  of  the  rules  that  are  often  passed  down  to  them  on  the  job  without  these  limitations  being 
specified.  Chamess  (1989)  points  out  that  the  novice  needs  to  be  warned  that  the  basic  principles 
have  side  conditions  attached.  Problems  of  oversimplification  can  be  prevented  when  exceptions 
and  limitations  are  included  with  the  rules.  Examples  of  situations  gathered  in  the  interviews  can 
be  used  to  teach  the  rules  in  context,  so  that  people  are  not  confused  on  how  to  follow  the  various 
rules  on  the  job. 

Structural  Knowledge 

Structural  knowledge  refers  to  the  organization  of  concepts  within  the  memory  system  of  an 
individual.  Knowledge  organization  and  the  relationship  of  the  elements  in  a  subject-matter 
domain  is  one  of  the  dimensions  that  may  underlie  the  components  of  proficiency  (Glaser,  Lesgold, 
&  Lajoie,  1987).  Structural  knowledge  is  not  only  essential  to  problem  solving  but  also  to  recall 
and  comprehension  of  information.  Memory  research  indicates  that  information  with  some 
structure  is  remembered  better  than  unstructured  lists  of  information  (Rumelhart,  1975; 
Thomdyke,  1977).  As  part  of  a  cognitive  task  analysis,  we  would  like  to  know  how  the  important 
concepts  in  a  domain  are  organized  by  experts,  so  that  these  concepts  can  be  taught  in  their 
relationship  to  each  other  rather  than  as  independent  entities. 

The  first  step  in  assessing  structural  knowledge  is  eliciting  knowledge  from  an  individual  or 
group  of  individuals.  Ratings  of  the  similarity  of  concepts  have  been  used  to  elicit  knowledge  in 
different  subject-matter  areas,  and  it  is  believed  to  be  a  method  independent  of  the  content  domain. 
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reliable  over  time  and  across  experts  (Jonassen,  Beissner,  &  Yacci,  1993).  The  next  step,  the 
analysis  and  representation  of  the  data,  is  a  process  of  inferring  the  psychological  structure.  These 
procedures  are  described  below. 

Eliciting  Structural  Knowledge 

Obtain  similarity  ratings  by  asking  individuals  to  rate  the  degree  of  similarity  between  pairs  of 
concepts.  This  method  identifies  a  group  of  important  concepts  and  has  the  respondent  rate  the 
degree  of  similarity  between  each  of  the  pairs  of  concepts. 

1.  Compile  a  list  of  the  most  important  concepts  in  the  domain.  These  concepts  could  relate 
to  the  entire  domain,  to  a  segment  of  it,  or  to  a  scenario.  No  more  than  24  concepts  should  be  used 
at  one  time  because  of  the  time  it  will  take  to  compare  each  concept  with  the  other  concepts. 

a.  Create  a  list  of  the  important  concepts  from  curriculum  and  instructor  guides  or  from 
the  terms  identified  as  relevant  to  a  scenario. 

b.  Have  two  or  three  subject  matter  experts  review  the  list  to  add  missing  concepts  or  to 
delete  unimportant  ones. 

c.  If  you  have  more  that  24  concepts,  have  two  to  three  subject  matter  experts  rate  the 
importance  of  each  concept  on  the  list  from  0  (not  important)  to  3  (most  important).  Select  from 
the  more  highly  rated  concepts.  Repeat  this  procedure  until  the  desired  number  of  concepts  is 
reached. 

2.  Using  the  selected  concepts,  create  a  randomized  list  of  all  possible  pairs  of  the  selected 
concepts. 

a.  Using  the  following  formula,  you  can  determine  the  number  of  pairs: 

x(x-n 

2 

where  x  is  the  number  of  concepts  used. 

For  example,  if  you  have  24  selected  concepts,  you  will  produce  276  possible  pairs. 

24(24-1)  =276 
2 

b.  Produce  a  randomized  list  of  all  possible  pairs.  It  would  be  helpful  if  you  had  a 
computer  program  that  did  this  for  you.  Someone  who  knows  some  elementary  computer 
programming  should  be  able  to  do  this. 

3.  Select  a  group  of  individuals  who  will  rate  the  list  of  all  possible  pairs  of  the  selected  con¬ 
cepts.  Depending  on  the  level  of  expertise  that  you  are  interested  in,  you  may  select  a  group  of 
experts,  intermediate  level  individuals,  and  novices.  For  example,  an  expert  rating  may  be  cre¬ 
ated  by  selecting  a  group  (2-5)  of  experts  that  are  extremely  knowledgeable  with  the  subject  mat¬ 
ter.  These  are  averaged  in  the  program  to  produce  the  expert  judgment. 
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4.  Ask  each  respondent  to  rate  the  strength  of  the  relationship  between  each  pair  of  concepts 
by  using  a  scale  of  1-7  where  1  indicates  a  very  weak  relationship  and  7  indicates  a  very  strong 
relationship  between  the  concepts.  For  276  comparisons,  it  takes  approximately  one-half  hour  to 
complete  the  ratings  of  similarity  of  24  concepts.  Present  a  simple  example  of  the  procedure: 

“Rate  how  closely  the  two  concepts  in  each  pair  are  related  to  each  other.” 

Yellow  Labrador— Cocker  Spaniel  [may  be  rated  a  7  (strong  relationship)] 

Yellow  Labrador— Pen  [may  be  rated  a  1  (weak  relationship)] 

Representing  Structural  Knowledge 

Description  and  example.  A  Pathfinder  Net  (PFNet)  is  a  graphic  network  in  which  concepts 
are  represented  inside  nodes  (rectangles)  and  the  relationships  between  the  concepts  are  depicted 
as  links  (lines)  that  connect  the  nodes  (Schvaneveldt,  1990;  Schvaneveldt  et  al.,  1985).  The  links 
are  given  a  value  that  reflects  the  strength  of  the  relationship  between  the  nodes.  These  values 
determine  the  length  of  the  path  from  one  concept  to  another.  Using  the  procedures  below,  a  PFNet 
can  be  produced  from  the  data  collected  and  described  above. 

Procedures.  The  procedure  is  automated  by  a  group  of  computer  programs  known  as 
Knowledge  Network  and  Orientation  Tool  (KNOT)  (Esposito,  McDonald,  Schvaneveldt,  &  Sitze, 
1992).  This  program  runs  on  the  Macintosh  computer.  The  process  is  as  follows: 

1.  Set  up  a  matrix  data  file  from  the  obtained  similarity  ratings.  Each  point  of  intersection  in 
the  matrix  indicates  the  respondent's  rating  of  similarity  between  those  two  concepts. 

a.  Using  any  text  editor,  write  the  concepts  both  in  the  rows  and  in  the  columns.  Fill  in 
each  point. 

Concept  A  B  C  D 

A  0  5  6  1 

B  5  0  3  2 

C  6  3  0  4 

D  12  4  0 

b.  Save  the  data  as  a  text  file. 

c.  Pathfinder  uses  two  parameters:  the  q  parameter  constrains  the  number  of  links 
traversed  in  paths  in  the  network;  the  r  parameter  defines  the  metric  used  for  computing  the 
distance  of  paths.  If  in  doubt,  the  q  parameter  in  the  KNOT  program  should  be  set  to  the  number 
of  concepts  minus  one,  and  r  should  be  set  to  infinity. 

2.  Analyze  the  matrix  using  the  Pathfinder  algorithm.  Open  a  data  file  under  Pathfinder  and 
the  system  automatically  does  all  of  the  computations.  This  produces  a  matrix  of  related  coeffi¬ 
cients.  The  computerized  program  uses  this  information  to  create  a  spatial  layout. 
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3.  Create  a  spatial  layout  using  the  Layout  function.  Figure  3  shows  the  output  of  the  Layout 
function  using  14  concepts  from  a  psychology  course.  Figure  4  shows  the  output  for  24  EW  con¬ 
cepts. 


Figure  3.  PFNet  for  psychological  concepts. 
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ait  -  ARH  missile 

asc  •  ASC  missile  ctaractoistics 

chf  -  chaff  solution 

elt  -  ELINT  parameter  limits 

eob  -  electronic  order  of  battle 

erk  •  emergency  racket  report 

fcr  -  fire  control  radars 

bat  -  hostile  attach  indicators 


jam -jamming 

onl  -  on-line  library 

pas  -  pass  down 

pit  -  platforms 

prf  -  pulse  rate  frequency 

reg  -  rtqiid  evaluation  guidelines 

set  -  scan  types 

sigs  -  signal  select 


shp  -  ship  speed 

sth  -  surface  threat  environment 

tar  -  target-acquisition-tracking  radars 

thb  -  threat  bearing 

tht  -  threat  terminology 

vam  -  vampire  report 

wan  -  warning  status 

wnd  -  wind  direction 


Figure  4.  PFNet  for  abbreviated  EW  concepts. 


4.  If  you  want  to  average  nets  from  several  respondents,  use  the  Average  function  that  is  part 
of  the  KNOT  program. 

5.  To  compare  two  nets  use  the  Similarity  function.  Similarity  is  determined  by  the  corre¬ 
spondence  of  links  in  two  networks.  The  measure  is  the  proportion  of  all  the  links  in  either  net¬ 
work  that  are  in  both  networks. 
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Training 


The  interconnection  of  elements  of  knowledge  has  been  identified  as  one  of  the  important 
components  of  competent  performance  (Glaser,  1989).  Once  these  interconnections  have  been 
identified,  PFNets  may  be  used  to  design  instruction.  Output  from  the  PFNet  program  would 
indicate  which  concepts  are  related  to  each  other  and  the  strength  of  the  relationship.  The  shorter 
the  line  between  two  concepts,  the  stronger  the  relationship.  When  any  of  the  concepts  are  first 
taught,  they  should  be  related  to  other  concepts  that  have  been  identified  as  connected  in  the  PFNet. 

PFNets  may  be  used  to  compare  the  knowledge  structures  that  are  present  for  most  experts  with 
those  of  an  individual  student  or  a  group  of  students.  Specifically,  the  Nets  would  indicate  the 
strength  of  relationship  between  two  concepts  for  the  expert.  If  there  were  no  link  or  a  very  weak 
link  for  the  student,  it  would  indicate  the  need  to  stress  more  information  and  experience  to 
strengthen  those  two  concepts.  In  addition,  the  knowledge  structures  of  students  can  be  compared 
at  the  start  and  end  of  a  course  to  look  for  changes  and  to  see  if  the  knowledge  structures  of  the 
students  at  the  end  of  the  course  are  more  similar  to  those  of  experts  than  they  were  at  the  beginning 
of  the  course. 


Conclusions 

The  methods  described  here  are  appropriate  for  conducting  a  cognitive  task  analysis  for  a 
complex  decision  making  task.  Any  of  the  methods  could  be  used  independently  or  in  combination. 
Development  of  a  task  process  model  is  time  consuming  but  is  important  for  obtaining  an 
organizing  framework  for  the  design  and  development  of  instruction.  The  information  flow  model 
is  appropriate  for  any  task  invoking  teams  or  individuals  who  communicate  with  each  other  in  the 
performance  of  the  task.  An  analysis  of  misconceptions  or  rules  of  thumb  will  help  define  the 
boundaries  of  any  rules  used  in  the  performance  of  a  task.  Structural  knowledge  analysis  involves 
little  effort  or  time  on  the  part  of  the  task  performers  but  does  involve  analyst  time  and  the  use  of 
a  computerized  program  for  its  execution. 

Recommendations 

We  recommend  that  the  cognitive  task  analysis  methods  described  here  be  used  in  revising 
instruction  for  a  complex  decision  making  task.  For  new  instruction,  these  methods  could  be  used 
in  conjunction  with  traditional  task  analysis. 
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Appendix  A 

Critical  Incident  Interview  Guide 


Critical  Incident  Interview  Guide 


A.  Describe  the  purpose  of  the  interview:  “We  want  to  learn  how  you  make  decisions  while 
working  on  your  job.  Please  respond  accurately  and  completely  and  take  your  time.  I  would  like  to 
use  the  tape  recorder  in  case  I  can't  write  everything  down,  if  that's  OK  with  you.  No  one  outside 
this  research  group  will  hear  the  tape.” 

B.  Give  the  interviewees  some  paper  and  a  pencil  in  case  they  want  to  make  a  diagram. 

C.  Start  the  interview. 

1.  Select  Incident. 

“I  would  like  you  to  discuss  some  difficult  decisions  or  events  that  occurred  while  you 
were  performing  your  job.  This  could  be  a  situation  where  an  error  was  likely,  or  did  in  fact  occur. 
Let’s  start  with  one  such  event.” 

2.  Obtain  unstructured  incident  account  in  the  form  of  a  time  line. 

“Describe  the  incident  from  start  to  completion.  I  would  like  to  reconstruct  the  events 
in  the  form  of  a  time  line  that  establishes  the  sequence  of  each  event.” 

Repeat  the  incident  back  to  the  interviewee  to  make  sure  you  have  it  correct. 

3.  Decision  point  identification. 

“Now  I  would  like  you  to  help  me  identify  specific  decision  points  on  the  time  line.  A 
decision  point  occurs  when  someone  else  might  have  chosen  to  act  differently.” 

4.  Decision  point  probing. 

For  each  decision  point,  use  the  following  probes. 

a.  Errors.  “If  an  error  occurred,  what  was  the  error?” 

Hypothetical.  “If  an  error  occurred,  how  should  the  incident  have  proceeded?” 

Missing  data.  “What  would  have  helped  you  make  the  decision?  What  data  or 
information  was  missing?” 

Error  avoidance.  “How  could  the  error  have  been  avoided?” 

b.  Cues/Situation  Awareness.  “What  were  you  seeing  or  hearing  that  caused  you  to 
make  this  particular  decision?” 

c.  Comparison  to  novice  (if  interviewing  an  expert).  “How  might  someone  with  less 
experience  have  responded?” 
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d.  Knowledge.  “What  information  did  you  use  in  making  this  decision,  and  how  was 
it  obtained?” 

e.  Decision  options. 

(1)  “What  other  choices  did  you  consider  or  were  available  to  you?” 

[If  none,  do  not  proceed  with  options.] 

List  other  choices: 

(a)  Restate  first  rejected  option. 

“What  was  the  reason  for  rejecting  it?” 

(b)  Restate  each  rejected  option  and  ask  the  reason  for  rejecting  it. 

(2)  Classification  of  the  types  of  options  elicited.  “Were  the  options  you  considered 

standard  or  constructed?” 

(a)  Standard—ones  you  are  commonly  taught  to  consider. 

(b)  Constructed— created  by  you  or  changed  from  the  standard  in  some  way. 
(If  so,  ‘What  experience  was  necessary  to  generate  this  option  [personal 
operator  experience,  special  training,  etc.]?”) 

f.  Rule  of  thumb.  “If  a  new  operator  was  standing  beside  you  during  this  incident  and 
asked  you  what  rule  (advice)  you  were  following  when  you  made  this  decision,  what  would  you 
say?” 


5.  Additional  incidents.  “Can  you  recall  another  incident  that  was  difficult  or  where  an 
error  might  likely  have  occurred?”  If  so,  repeat  steps  2-5. 
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Appendix  B 

Example  of  Questions  Used  to  Complete 
the  Task  Process  Model 
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Example  of  Questions  Used  to  Complete 
the  Task  Process  Model 

Set  (Preparation) 

1.  When  you  start  to  stand  watch,  what  ate  you  looking  for? 

Initiating  Cues 

2.  What  do  you  do  when  a  major  alert  is  heard? 

3.  "What  do  you  do  during  a  lull  when  no  new  emitters  are  coming  up? 

Emitter  Identification 

4.  When  an  emitter  first  comes  up,  what  knowledge,  sources,  or  resources  do  you  use  in  eval¬ 
uating  it?  In  what  order? 

5.  Do  you  use  Rapid  Evaluation  Guidelines?  How  do  you  use  them?  At  what  point  do  you 
use  them? 

6.  Do  you  choose  to  listen  to  the  emitter  signal  very  often?  At  what  point  do  you  listen  to  the 
signal? 

7.  Do  you  use  the  on-line  library  much?  When  do  you  use  the  on-line  library? 

8.  If  there  are  two  or  more  missiles  on  the  screen  at  approximately  the  same  time  and  the 
computer  system  is  too  slow,  what  do  you  do? 

9.  WHiat  do  you  do  when  an  unknown  emitter  appears  on  the  screen? 

10.  What  do  you  do  when  a  friendly  or  neutral  emitter  appears  on  the  screen? 

11.  Do  you  log  friendly  emitters?  As  soon  as  you  recognize  them  or  at  a  lull? 

12.  At  what  point  do  you  report  missile  composition? 

Information  Dissemination 

13.  If  you  are  not  certain,  should  you  report  a  suspected  missile? 

14.  Do  you  report  a  missile  if  you  cannot  identify  it?  Or  do  you  assume  someone  else  in  the 
battle  group  will  identify  it? 

15.  At  what  point  do  you  report  a  threat? 

16.  At  what  point  do  you  report  a  missile  to  the  officer  in  charge?  When  do  you  report  a  mis¬ 
sile  over  the  net? 


B-1 


17.  At  what  point  do  you  report  launching  countermeasures?  To  whom  do  you  report? 

Countermeasures 

18.  Under  what  circumstances  does  the  watch  supervisor  take  over  equipment  operation? 

19.  Under  what  circumstances  do  you  use  countermeasures?  Describe  how  you  use  it. 

20.  Under  what  circumstances  would  you  not  take  countermeasures? 

21.  If  two  or  more  missiles  appear  close  together  in  time,  what  countermeasures  do  you  take? 

22.  What  elements  do  you  consider  when  determining  what  countermeasures  to  use? 

23.  After  using  countermeasures,  how  do  you  determine  if  the  countermeasure  has  been  suc¬ 
cessful? 

24.  If  a  missile  is  inactive  after  using  countermeasures,  do  you  sequence  through  to  delete  the 
missile  from  the  screen? 
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Appendix  C 

Task  Process  Model  for  the  EW  Task 
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